Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Can Incoming Interns Be Entrusted to Recognize Medical Emergencies? Implementation of a Vignette-Based Cross-Cover Assessment.

Background: Interns must recognize urgent clinical situations and know when to seek assistance. However, assessing this skill is challenging. Objective: We explored whether graduating medical students could determine urgency of medical cross-cover scenarios and what factors were associated with this ability. Methods: Sixty senior medical students enrolled in an internal medicine residency preparation course, and 28 experts were invited to take an assessment using 4 clinical vignette handoffs, each with 5 to 6 cross-cover scenarios. Respondents were asked whether they would evaluate the patient at bedside and notify their supervising resident. They were asked to rate their comfort managing the scenario, rate the urgency (1=low, 2=moderate, 3=high), and take a medical knowledge quiz. Student performance was categorized based on stratification of clinical urgency-those who underestimated (fourth quartile), accurately estimated (second and third quartile), and overestimated (first quartile) urgency. We examined differences between groups in medical knowledge, action, and confidence using analysis of variance and post-hoc Tukey Honestly Significant Difference test. Results: Fifty-eight students (96.7%) and 22 experts (78.6%) participated. Clear differentiation emerged between students' ability to estimate urgency on the 3-point urgency scale (lowest quartile: 2.15±0.11; mid-quartiles: 2.38±0.07; upper quartile: 2.61±0.10, respectively). Students who underestimated urgency were less likely to notify their supervising resident (P=.001) and less likely to evaluate a patient at bedside (P=.01). There was no difference in quiz score or comfort level. Conclusions: Incoming interns vary in their abilities to recognize urgent scenarios, independent of medical knowledge and confidence.

2015 American College of Rheumatology Workforce Study and Demand Projections of Pediatric Rheumatology Workforce, 2015-2030.

OBJECTIVE: To describe the character and composition of the 2015 pediatric rheumatology workforce in the US, evaluate current workforce trends, and project future supply and demand of the pediatric rheumatology workforce through 2030. METHODS: The American College of Rheumatology created the workforce study group to study the rheumatology workforce. The workforce study group used primary and secondary data to create a representative workforce model. Pediatric rheumatology supply and demand was projected through 2030 using an integrated data-driven framework to capture a more realistic clinical full-time equivalent (FTE) and produce a better picture of access to care issues in pediatric rheumatology. RESULTS: The 2015 pediatric rheumatology workforce was estimated at 287 FTEs (300 providers), while the estimated excess demand was 95 (33%). The projected demand will continue to increase to almost 100% (n = 230) by 2030 if no changes occur in succession planning, new graduate entrants into the profession, and other factors associated with the workforce. CONCLUSION: This study projects that the pediatric rheumatology workforce gap will continue to worsen significantly from the 2015 baseline, and by 2030 the demand for pediatric rheumatologists will be twice the supply. Innovative strategies are needed to increase the workforce supply and to improve access to care.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education.

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy.

BACKGROUND: Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. METHODS: In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels. RESULTS: In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256). DISCUSSION: For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

What is the association between student well-being and high-stakes examination scores?

INTRODUCTION: As educators seek to improve medical student well-being, it is essential to understand the interplay between distress and important outcomes. Performance on Step 1 of the United States Medical Licensing Examination has played a significant role in selection for postgraduate residency positions in the United States and consequently has been a source of great stress for medical students. The purpose of this study was to examine whether student well-being correlates with performance on a high stakes licensing examination. METHODS: Between 2014 and 2016, three sequential cohorts of medical students at the University of Michigan Medical School completed the Medical Student Well-Being Index (MSWBI) at the end of their 2nd-year curriculum, shortly before taking Step 1. Associations between well-being and Step 1 scores were investigated while adjusting for MCAT scores and cumulative second-year course scores. RESULTS: In total, 354 students were included in the analysis (68.1% of potential responders). On bivariate analysis, poor student well-being (0 = low distress [high well-being], 7 = high distress [poor well-being]) was associated with lower Step 1 examination scores (slope = -2.10, P < .01), and well-being accounted for 5% of overall Step 1 score variability (R2  = .05). However, after adjustment for MCAT scores and cumulative GPA (full model R2  = .51), the relationship between well-being and Step 1 score was no longer significant (slope = -0.70, P-value = .06). CONCLUSIONS: When controlling for metrics of academic performance, student well-being prior to taking Step 1 was not associated with how well students performed on Step 1 for the study sample.

Managing expansions in medical students' clinical placements caused by curricular transformation: perspectives from four medical schools.

Many challenges could occur that result in the need to handle an increase in the number of medical student clinical placements, such as curricular transformations or viral pandemics, such as COVID 19. Here, we describe four different institutions' approaches to addressing the impact of curricular transformation on clerkships using an implementation science lens. Specifically, we explore four different approaches to managing the 'bulge' as classes overlap in clerkships Curriculum leaders at four medical schools report on managing the bulge of core clinical placements resulting from reducing the duration of the foundational sciences curriculum and calendar shifts for the respective clerkship curriculum. These changes, which occurred between 2014 and 2018, led to more students being enrolled in core clinical rotations at the same time than occurred previously. Schools provided respective metrics used to evaluate the effectiveness of their bulge management technique. These data typically included number of students affected in each phase of their curricular transformation, performance on standardized examinations, and student and faculty feedback. Not all data were available from all schools, as some schools are still working through their 'bulge' or are affected by COVID-19. There is much to be learned about managing curricular transformations. Working on such endeavors in a learning collaborative such as the AMA Accelerating Change in Medical Education Initiative provided support and insights about how to survive, thrive and identifying lessons learned during curricular transformation.

Medical Student Attitudes toward USMLE Step 1 and Health Systems Science - A Multi-Institutional Survey.

Phenomenon: Because of its importance in residency selection, the United States Medical Licensing Examination Step 1 occupies a critical position in medical education, stimulating national debate about appropriate score use, equitable selection criteria, and the goals of undergraduate medical education. Yet, student perspectives on these issues and their implications for engagement with health systems science-related curricular content are relatively underexplored. Approach: We conducted an online survey of medical students at 19 American allopathic medical schools from March-July, 2019. Survey items were designed to elicit student opinions on the Step 1 examination and the impact of the examination on their engagement with new, non-test curricular content related to health systems science. Findings: A total of 2856 students participated in the survey, representing 23.5% of those invited. While 87% of students agreed that doing well on the Step 1 exam was their top priority, 56% disagreed that studying for Step 1 had a positive impact on engagement in the medical school curriculum. Eighty-two percent of students disagreed that Step 1 scores should be the top item residency programs use to offer interviews. When asked whether Step 1 results should be reported pass/fail with no numeric score, 55% of students agreed, while 33% disagreed. The majority of medical students agreed that health systems science topics were important but disagreed that studying for Step 1 helped learn this content. Students reported being more motivated to study a topic if it was on the exam, part of a course grade, prioritized by residency program directors, or if it would make them a better physician in the future. Insights: These results confirm the primacy of the United States Medical Licensing Examination Step 1 exam in preclinical medical education and demonstrate the need to balance the objectives of medical licensure and residency selection with the goals of the broader medical profession. The survey responses suggest several potential solutions to increase student engagement in health systems science curricula which may be especially important after Step 1 examination results are reported as pass/fail.

Training Medical Students to Create and Collaboratively Review Multiple-Choice Questions: A Comprehensive Workshop.

Introduction: Multiple-choice question (MCQ) creation is an infrequently used active-learning strategy. Previous studies demonstrated that medical students find value in the process, but have minimal training, which may limit potential learning benefits. We therefore developed a process for question-creation that required students to complete in-depth training, in addition to collaborative question-writing and editing. Methods: We created a question-writing workshop consisting of three components: (1) training in MCQ writing utilizing NBME online modules, a practice MCQ-writing session, and a training session, (2) writing MCQs independently after choosing topics from an institutionally generated blueprint, and (3) reviewing and editing MCQs via an in-person session. To understand students' perceptions, we held two four-student focus groups and recorded/transcribed the data. We iteratively reviewed the transcripts to generate a codebook and corresponding themes. We used the focus group data to generate a survey with Likert-scale questions, which we sent to the remaining 10 students and analyzed using Microsoft Excel. Results: Eighteen second-year medical students participated in this workshop. Students perceived that question-writing training (3.7/5.0±0.5) and question writing (3.9/5.0±0.3) benefitted their learning. Students perceived that MCQ writing required concept integration (4.1/5.0±0.6). Students described how question writing allowed them to recognize subtle distinctions between therapies and diagnoses. Each MCQ required about 1.5 hours to write and collaboratively edit. Discussion: Our results demonstrated that students perceived question writing to benefit their learning. More importantly, students felt that question writing actively engaged them to integrate content and compare concepts; students' engagement suggests that they learned from this question-writing activity.

Competency Committees in Undergraduate Medical Education: Approaching Tensions Using a Polarity Management Framework.

Implementing competency-based medical education in undergraduate medical education (UME) poses similar and unique challenges to doing so in graduate medical education (GME). To ensure that all medical students achieve competency, educators must make certain that the structures and processes to assess that competency are systematic and rigorous. In GME, one such key structure is the clinical competency committee. In this Perspective, the authors describe the University of Michigan Medical School's (UMMS's) experience with the development of a UME competency committee, based on the clinical competency committee model from GME, and the first year of implementation of that committee for a single cohort of matriculating medical students in 2016-2017.The UMMS competency committee encountered a number of inter dependent but opposing tensions that did not have a correct solution; they were "both/and" problems to be managed rather than "either/or" decisions to be made. These tensions included determining the approach of the committee (problem identification versus developmental); committee membership (curricular experts versus broad-based membership); student cohort makeup (phase-based versus longitudinal); data analyzed (limited assessments versus programmatic assessment); and judgments made (grading versus developmental competency assessment).The authors applied the Polarity Management framework to navigate these tensions, leveraging the strengths of each while minimizing the weaknesses. They describe this framework as a strategy for others to use to develop locally relevant and feasible approaches to competency assessment in UME.

Creating assessments as an active learning strategy: what are students' perceptions? A mixed methods study.

Background: Teaching students how to create assessments, such as those involving multiple-choice questions (MCQs), has the potential to be a useful active learning strategy. In order to optimize students' learning, it is essential to understand how they engage with such activities. Objective: To explore medical students' perceptions of how completing rigorous MCQ training and subsequently writing MCQs affects their learning. Design: In this mixed methods exploratory qualitative study, eighteen second-year medical students, trained in MCQ-writing best practices, collaboratively generated a question bank. Subsequently, the authors conducted focus groups with eight students to probe impressions of the process and the effect on learning. Responses partially informed a survey consisting of open-ended and Likert rating scale questions that the remaining ten students completed. Focus group and survey data from the eighteen participants were iteratively coded and categorized into themes related to perceptions of training and of collaborative MCQ writing. Results: Medical students felt that training in MCQ construction affected their appreciation for MCQ examinations and their test-taking strategy. They perceived that writing MCQs required more problem-solving and content-integration compared to their preferred study strategies. Specifically, generating plausible distractors required the most critical reasoning to make subtle distinctions between diagnoses and treatments. Additionally, collaborating with other students was beneficial in providing exposure to different learning and question-writing approaches. Conclusions: Completing MCQ-writing training increases appreciation for MCQ assessments. Writing MCQs requires medical students to make conceptual connections, distinguish between diagnostic and therapeutic options, and learn from colleagues, but requires extensive time and knowledge base.

Workforce Trends in Rheumatology.

The United States is facing a rheumatology provider shortage over the next decade, which will negatively affect care for patients with rheumatic disease across the nation if this deficit is not thoughtfully addressed. The increasing numbers of retiring rheumatology specialists, women entering the workforce, and rheumatology graduates seeking part-time employment were identified as the most significant factors driving the projected decline in supply of providers. The major factors driving the projected increase in demand include an aging and growing population and improved treatment options, both of which increase disease prevalence and the challenge of managing chronic rheumatologic diseases.

Employing a Root Cause Analysis Process to Improve Examination Quality.

PROBLEM: Multiple-choice question (MCQ) examinations represent a primary mode of assessment used by medical schools. It can be challenging for faculty to produce content-aligned, comprehensive, and psychometrically sound MCQs. Despite best efforts, sometimes there are unexpected issues with examinations. Assessment best practices lack a systematic way to address gaps when actual and expected outcomes do not align. APPROACH: The authors propose using root cause analysis (RCA) to systematically review unexpected educational outcomes. Using a real-life example of a class's unexpectedly low reproduction examination scores (University of Michigan Medical School, 2015), the authors describe their RCA process, which included a system flow diagram, a fishbone diagram, and an application of the 5 Whys to understand the contributors and reasons for the lower-than-expected performance. Using this RCA approach, the authors identified multiple contributing factors that potentially led to the low examination scores. These included lack of examination quality improvement (QI) for poorly constructed items, content-question and pedagogy-assessment misalignment, and other issues related to environment and people. OUTCOMES: As a result of the RCA, the authors worked with stakeholders to address these issues and develop strategies to prevent similar systematic issues from reoccurring. For example, a more robust examination QI process was developed. NEXT STEPS: Using an RCA approach in health care is grounded in practice and can be easily adapted for assessment. Because this is a novel use of RCA, there are opportunities to expand beyond the authors' initial approach for using RCA in assessment.

Does Reducing Clerkship Lengths by 25% Affect Medical Student Performance and Perceptions?

PURPOSE: Transforming a medical school curriculum wherein students enter clerkships earlier could result in two cohorts in clerkships simultaneously during the transition. To avoid overlapping cohorts at the University of Michigan Medical School, the length of all required clerkships was decreased by 25% during the 2016-2017 academic year, without instituting other systematic structural changes. The authors hypothe sized that the reduction in clerkship duration would result in decreases in student perfor mance and changes in student perceptions. METHOD: One-way analyses of variance and Tukey post hoc tests were used to compare the 2016-2017 shortened clerkship cohort with the preceding traditional clerkship cohorts (2014-2015 and 2015-2016) on the following student outcomes: National Board of Medical Examiners (NBME) subject exam scores, year-end clinical skills exam scores, evaluation of clerkships, perceived stress, resiliency, well-being, and perception of the learning environment. RESULTS: There were no significant differences in performance on NBME subject exams between the shortened clerkship cohort and the 2015-2016 traditional cohort, but scores declined significantly over the three years for one exam. Perceptions of clerkship quality improved for three shortened clerkships; there were no significant declines. Learning environment perceptions were not worse for the shortened clerkships. There were no significant differences in performance on the clinical skills exam or in perceived stress, resiliency, and well-being. CONCLUSIONS: The optimal clerkship duration is a matter of strong opinion, supported by few empirical data. These results provide some evidence that accelerating clinical education may, for the studied outcomes, be feasible.

Early Diagnosis and Treatment of Rheumatoid Arthritis.

Rheumatoid arthritis is the most common inflammatory arthritis, and a significant cause of morbidity and mortality. Primary care providers should be able to distinguish the clinical presentation of rheumatoid arthritis from osteoarthritis, because the treatment and outcomes differ greatly between these 2 common forms of arthritis. This article provides a current overview of our understanding of rheumatoid arthritis, with an emphasis on early diagnosis and approaches to treatment.